COORDINATED AUDIO AND VISUAL OMNIDIRECTIONAL RECORDING 



FIELD OF THE INVENTION 

This invention relates to simultaneous recording of panoramic visual data and panoramic 
audio dat and the application of the directed audio (or radiated) signal to focus the visual 
recording on the source of the directed audio signal. 
BACKGROUND AND INFORMATION DISCLOSURE: 

The use of video photography has evolved steadily over the past forty years in its 
application to surveillance situations and environments. Applications include stationary 
systems such as are found in commercial establishments, and on moving bases such as 
robots, on submarines, vehicles, etc. 

For example, U.S. Patent 6.016,385 to Yee et al discloses a real time remotely controlled 
robot having a head with a pair of television cameras (eyes) and a pair of microphones 
(ears) and a pair of arms mounted on a mobile trunk, an operated located at a command 
center receives audio and visual signals from the robot and, in response, issued commands 
that control movement of the trunk and arms to perform tasks. 

U. S. Patent 4,453,085 to Pryor discloses an electro-optical system for monitoring the 
positioning of a movable arm. 



U. S. Patent 4,604,559 To Hawkes et al discloses a robot controlled by a user wherein an 
audio signal is fed back from the robot to the user regarding force and movement 
parameters of the robot operation. 

U. S. Patent 5,299,288 to Glassman et al discloses an image directed robot system for 
surgical applications including an optical tracking camera supplying data to the controlling 
processor. 

One of the major developments in video photography in recent years has been the 
panoramic camera. According to this technology a lens with a 360° field of view stores 
the entire field in memory and various areas are selected and projected according to the 
interests of the viewer. This technique has many applications, for example, in real estate 
advertisements where features of the location are selected and enlarged for detailed 
examination by the viewer. 

U.S. Patent 5,920,337 to Glassman et al discloses an omnidirectional visual image detector 
and processor (incorporation herein by reference). There is described a reflective round 
lens (either a cone or spherical section) which reflects the object beam from a surrounding 
view (encircling the lens) to a direction parallel to the axis of the rotund lens. The beam 
then passes through an objective lens and is focussed onto a CCD (charge coupled device) 
where image data is detected and processed for storage or image presentation on a monitor. 



In the context of this specification, the term, "visually examine" means that, a narrow field 
of view is selected covering a few degrees from an entire 360° field of view recorded by a 
panoramic video camera. 

SUMMARY OF THE INVENTION. 

It is an object of this invention to develop a surveillance device that enables a user to 
visually examine a location that has a source of a signal. The signal is an audial signal , in 
one version or an electr omagnetic signal (radio, microwave, light, infrared, etc) in another 
version. . 

This invention is directed toward an omnidirectional camera having a lens for viewing a 
360° field of view; a visual memory for storing visual data at addresses representing the 
field of view; an omnidirectional detector for detecting the direction of location of signal 
source, A display selection circuit is provided for selecting visual data from an address in 
the visual memory corresponding to the direction of location. The visual data is displayed 
on a system monitor. 

In one version, a directional audio detector and a panoramic CCD (charge coupled device) 
camera installed in the center of a site, (e.g., store, prison yard, battlefield, etc.) The camera 
is continually generating visual data of a 360° view of the site. If a sound is generated 
from a location in the site, ( an intruder, a shout, a shot, etc.) then the audio receiver of this 
invention will detect the sound and its direction relative to the camera and direct the camera 



to display that portion of the stored data representing the visual image of the source. 

Another object of the invention is to provide a panoramic audio detector that is capable of 
receiving an audio signal from a source located any where on a 360° circumference around 
the audio detector, "point" to the location of the source of sound and then amplify the audio 
signal for more detailed examination of the sound. The data corresponding to the 
direction of the source of sound is used to control the display selected from the visual data 
stored by the camera. 

In one embodiment, the panoramic audio detector of this invention comprises a plurality of 
audio detectors, strung one after the other on a circle. Each audio detector comprises a 
piezo electric element, a flip-flop, and coupling circuitry between the piezo electric element 
and its respective flip-flop. The coupling circuitry enables the piezo element to set its 
respective flip-flop at the instant a sound impulse is received by the piezo-electric. The 
direction of the source of sound will determine which flip-flop is set before all the other 
flip-flops. The first flip-flop to be set will store its address (referred to herein as the 
"source address") in a central register and the central register will ignore signals from all 
other flip flops during the sampling period. 

The register will thereby have stored an address corresponding to the direction of the 
source. This address is used to select camera data from the panoramic camera for 
localized display of the source of sound 




The phase of the sound signal reaching the detector closest to the source will be ahead of 
the phases of the same sound signal reaching the other (more distal) detectors. The 
invention includes an audio "focusing" circuit that adjusts the phase of the signal from each 
detector so that each signal from the respective detector is in phase with the detector nearest 
the source of sound. An adder adds together all of the phase adjusted signals to generate a 
"summation sound" signal from the source. 

In another embodiment, a focusing circuit coupled to the panoramic camera displays the 
image of a location in response to motion in that location. The panoramic viewing system 
has two video memories, and a comparator that is configured to compare point by point 
image data at each memory address of one visual memory to data at the corresponding 
addresses of the other visual memory. A difference between a signal stored at one address 
in one memory and the signal at the same address in the other memory indicates motion at 
the location and this difference causes the circuit to display the image of the location where 
motion has occurred. . 

BRIEF DESCRIPTION OF THE FIGURES: 

Fig. 1 shows an audial omnidirectional detector of this invention. 

Fig. 2A and 2B show one embodiment of a selection means. 

Fig. 3 is a timing diagram of the emboeiment of fig. 4. 

Fig. 4 shows another emboeiment of a selection technique. 

Fig, 5 Show an audial array having two detectors. 

Fig. 6 shows a version of the invention being a panoramic teleconference table. 
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DESCRIPTION OF ILLUSTRATIVE EXAMPLES: 

Turning to a discussion of the drawings, fig. 1 is a schematic diagram of one version of the 
invention 10 in which the directional function of the panoramic camera responds to a sound 
coming from a source at a location on the circumferential field of view. 

There is shown an omnidirectional visual imaging system 13 including an array of "sub " 
lenses 12, focussing lens 14 and charge coupled device array 16. The omnidirectional 
visual imaging system 13 is of the type disclosed by U.S.Patent 5,920,337 to Glassman 
which is incorporated herein by reference. 

An audio detector system 1 1 is shown as a string of detectors 30 mounted on a circle 
that is coaxial with an axis 18 of the lens 12. The detectors 39 are preferably piezo electric, 
capacitive, semiconductor elements or similar pressure sensitive devices well known in the 
art . 

Each detector 30 is coupled to a selection circuit 49 that responds to the first detector to 
receive a sound wave radiated from a source 34 by shutting off the signals from all of the 
other detectors. 

Sound generated by a source 34 from a location on the circumferential field of view 
reaches first the detector 30A closest to the source 34 before it reaches the other detectors 
thereby shutting off the signals from all of the other detectors. An address register 36 
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registers the address of the closest detector 30A and ignores the signals from the remaining 
audial detectors 32. The registered address is applied to display control circuit 38. The 
visual display circuit 38 selects the visual data from the corresponding address in the video 
memory 40 and displays an image of the location on the system monitor 42. 

Also displayed is a number indicating the azimuthal direction of the source of sound 
relative to the camera. 

One embodiment of a selection circuit 49 is shown in fig. 2.A. showing an array (ring) of 
audial detectors 30 . Fig. 2B shows a portion of circuit 49 to better advantage than fig. 2A. 
The output terminal of each audial detector 30 is connected to an input terminal of an AND 
gate 5 1 . Each detector 30 has its own AND gate 51 . 

The output terminals of all AND gates 51 have two connections: 

(i) One connection 63 is to an address register (not shown) which activates a portion of the 
CCD to send an image signal representing that portion of the field of view to a video 
monitor. 

(ii) The second connection of all output terminals 53 of all the AND gates 51 is connected 
to a bus 57. Bus 57 is connected to an input terminal of a single inverter 55 whose output 
terminal is connected to bus 59. Bus 59 is connected to a second input terminal 61 of all 
AND gates 51. 



Initially there is no sound incident on any of the detectors so that the input signal to all of 
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the gates 51 is zero. When a sound signal is radiated from a sound source, the closest 
detector is the first to receive the signal and sends a positive signal to one input of its gate 
Since the signal on the second input terminal is zero (not negative) the first gate to receive 
the signal from its detector 30 will send a signal to the address register 63. The positive 
signal from the first gate to receive the sound signal is inverted by inverter 55 and the 
negative signal is applied through the second bus 59 to the second input terminal of all the 
remaining gates thereby turning off all remaining gates. 

The video memory 40 reads the selected address from register 63 and applies the image 
data on monitor 42. 

An alternate scheme for detecting the direction of the sound is shown in fig. 4. There is 
shown a ring of audial detectors 30 arranged relative to the omnidirectional lens. The lens 
is not shown in fig.4 but is arranged coaxially with the audial detector array as discussed 
with regard to fig. 1. 

Each detector 30 of the array of detectors forms a pair of detectors with one member of 
each pair being positioned on a circumferential boundary oppositely from another 
member of the pair. For example, detectors 30A and 30B form one pair, detectors 30C 
and 30D form another pair A plurality of difference circuits 37 is provided although only 
two difference circuits are shown in fig. 4. a difference circuit for each pair of detectors, 
one pair of detectors for each difference circuit 37. 

Fig. 4 shows further: a inverters 39 (two are shown), 



one inverter for each pair of detectors, one pair of detectors for each inverter; 

a rectifiers 49, one rectifier 49 for each pair of detectors 30, one pair of detectors 30 for 
each rectifier 49; 

an address register 41 having a plurality of address input terminals, one address terminal 
for one difference circuit and one difference circuit for one address terminal; 

one member of each pair of detectors 30 is connected to transmit an image signal to an 
input terminal of one of the inverters 39; 

an output terminal of the inverter is connected to an input terminal of one of the difference 
circuits; 

another one of each pair of detectors is connected to another input terminal of its difference 
circuit; 

each one of the rectifiers 49 has an input terminal connected to an output terminal of the 
difference circuit 37; 

each one of the rectifiers 49 has an output terminal connected to one of the input terminals 
of the address register 41 . 
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the address register has an output terminal configured to emit an address signal responsive 
to a null signal from one of the rectifiers. 

The timing diagrams of figs. 3.1-- 3.7 show the signals generated by a sound pulse 46 
incident on the array of detectors 30. Detector 30C is the "leading" detector where the 
phase shift of the incoming wave will be assumed to be zero as shown in fig. 3,4. Figs. 
3.1 and 3.2 show that the signals received by detectors 30A and 30B have shifted an equal 
amount, 0, so that the difference between their signals (fig. 3,3) is zero. The inverted 
signal, Fig. 3.5, from detector 30D has shifted by 20 thereby giving pulse 3.6. 

Fig. 3.7 is a plot of the peak output of each difference circuit 37 vs. 0. Interpolation of the 
plot to Zero yields the value of 0 being the direction of the sound 46. 

When no sound is incoming, the output S3 = 0 (fig. 3.7) . When a sound wave 46 is 
incident on the array 30, then a finite pulse 3.7 is generated, from the respective one of the 
difference circuits that stores that the signal in the register at its azimuthal address 0 
Selection circuit 45 then applies the address Q to video memory 46 to locate the video data 
corresponding to the source of the sound 46. Video memory 46 is connected to CCD 
array 16 for storing an image signal generated by the CCD 36 responsive to the visual 
image of the surrounding field formed by omnidirectional lens 13. The selected image data 
is presented on video monitor 42. Not shown in fig. 2 but shown in fig. 1, the axis of the 
audial array is coincident with the axis of the omnidirectional lens 13. 

Fig. 5 shows another embodiment of the invention being a system which relies on the 
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phase difference of the sound wave between a pair of audial detectors 50 and 52. to present 
on a video monitor 42 an image of an area of a field of view formed by an omnidirectional 
lens 13 on a CCD array and stored in video memory 40. 

If P represents the period between the arrival of the earliest detectable portion of a sound 
signal, (such as a shot) to each detector. The velocity of sound is C. Then the angle, 0, 
of the direction of the sound relative to the audio detector line is 

cos0 = rc /D 

In practice, visual data is stored in the visual memory 40 at addresses corresponding to 
the angle 0. The phase detector 51 measures the time period P being the difference in 
times of arrival of the signal from source S to first audial detector 50 and detector 352. 
The value of 0, calculated by the formula, gives the address of the image data for 
generating a local image to be displayed directly on the monitor 42. 

Several embodiments have been described of a surveillance system being a combination of 
audial detectors that sense the direction of an incoming sound wave and a visual system 
that presents an image of the area includes the source of the sound. 

Other variations and modification of this invention may be contemplated after reading the- 
specification and studying the drawings that are within the scope of the invention. 

For example other forms of lens (other than an omnidirectional lens may be used to collect 
the image data 
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Fig, 6 shows an embodiment for application in televising a conference. The conferees 62 
(seven are shown)are located (seated) at locations around a panoramic camera 64 
positioned on a support 65 (e.g., table). Each location has a microphone 66. Each 
location is viewed through a corresponding section of the lens of camera 64. When the 
voice of the conferee 62 nearest a particular microphone 64 at the conferee's location is 
detected by the microphone 64, the voice signal is amplified and transmitted to signal 
selection controller 70 which conditions the panoramic camera 64 to project the image 
formed by the corresponding section 68 on the monitor 72. The signal selection controller 
70 is shown in phantom in fig. 6 because it is preferably located under the table 65. 

In another arrangement, the signal selection means includes includes an array of buttons 
(switches) on a console 73. Each switch, when closed, connects a selected one of the , 
microphones 62 to a source of power (not shown) enabling the selected microphone 62 to 
convert sound to an electrical signal for transmission to the monitor, coincident with 
selection of the corresponding section of the field of view for viewing on the monitor. In 
this arrangement, selection of the microscope occurs simultaneously with the selection of 
the video signal for viewing corresponding to the selected microphone. 

In other versions: 

the system is mountable on a cart or robot and transportable into an otherwise inaccessible 
environment. 
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the principles of the invention are adaptable to underwater surveillance. 

the system may be adapted to use at night by using a visual system that is sensitive to 
infrared 

The audial detector in various embodiments include piezoelectric detectors, capacitive 
detectors and semiconductor detectors such as are well known in the art. 

I therefore wish to define the scope of the invention by the appended claims. 
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